ARC596 - Tutorial 3 - Unity AR App

ARC596 - Tutorial 3 - Unity AR App

• ARC596: Embodied Computation
• Professor: Daniela Mitterberger - mitterberger@princeton.edu
• Assistant Instructor: Kirill Volchinskiy - kvolchinskiy@princeton.edu

Requirements

1. Rhinoceros 7
2. Github Desktop
3. Anaconda
4. Unity 2022.3.3f1

   *Note: Android SDK and Java JDK (when developing for Android) - have to be ticked in the installation modules when installing Unity.*
   

Dependencies

1. COMPAS
2. COMPAS Fab - Fabrication Library for Robots
3. COMPAS Eve - Communication
4. Vuforia
5. ROS#

Create an AR Test App

• Open the file you created when following the Installation Guide using Unity Hub on your computer or use the animation app you just made and delete the existing GameObjects.

Note: Be sure that the AR dependencies are installed per the prior section of this tutorial.

Configure an AR session and add AR Foundation components

A scene needs an AR session to enable AR processes, such as motion tracking, environmental understanding, and lighting estimation. You will need the following game objects to support an AR session:

• AR Session: Controls the lifecycle of an AR experience.
• XR Origin: Transforms AR coordinates into Unity world coordinates.

1. Before adding the new game objects, delete the default Main Camera. It will be replaced by a new Camera Offset in the XR Origin.

   

2. Add the new AR game objects to your scene: right-click the Hierarchy pane and select XR. Add a new AR Session and a new XR Origin (Mobile AR) game object.

   

What is a Session?

All AR processes, such as motion tracking, environmental understanding, and lighting estimation, happen inside an ARCore session. ARSession is the main entry point to the ARCore API. It manages the AR system state and handles the session lifecycle, allowing the app to create, configure, start, or stop a session. Most importantly, it enables the app to receive frames that allow access to the camera image and device pose.

• Your Hierarchy should now look like this:

  

• Expand the XR Origin you created in the Hierarchy, and select the Camera Offset. In the inspector, change its Tag to MainCamera.

Detect planes in the real world

Add an ARPlane Manager Component and place an AR Plane Prefab

An ARPlaneManager detects ARPlanes and creates, updates, and removes game objects when the device’s understanding of the environment changes.

• Go to Hierarchy and click on the XR Origin GameObject. On the Inspector Window, click on Add Component, search** for the AR Plane Manager and Add it.

  

• Respectively, click again on the Add Component, search for AR Raycast Manager and add it as well.

  

We will need this component later. It helps obtain information of the raycasts deriving from the user input on the screen.

• In the Hierarchy Window, right click , go to XR > AR Default Plane. This will create an AR-configured plane, that we will customize a bit and use as a Prefab for our application, in order to visualize the AR detected planes.

  

This XR GameObject contains scripts that generate and visualize the planes detected by the AR device, using the AR Plane manager script we created in the previous step.

• AR Plane script summary: Represents a plane (i.e., a flat surface) detected by an AR device. It is generated by the ARPlaneManager when an AR device detects a plane in the environment.

• AR Plane Mesh Visualizer Summary: Generates a mesh for an <ARPlane>. It generates a mesh and updates the boundary points with a <LineRenderer>.

• Now we will go to the Project Window, create a new empty folder in our Assets, and name it “Prefabs”, to organize our material. (Right click > Create > Folder). Then, double click it and go inside the folder.

  

• Drag and drop the AR Default PlaneGameObject inside the Prefabs folder, in order to save it as a _Prefab. In general, this is an easy way to create new Prefabs, after we have edited them as we like in our Scene.

  

Note: In short, Unity’s Prefab system allows you to create, configure, and store a GameObject complete with all its components, property values, and child GameObjects as a reusable Asset. The Prefab Asset acts as a template from which you can create new Prefab instances in the Scene.
→ More info about prefabs

→ Notice that the AR Default Plane becomes blue in the Hierarchy. This is because the process of creating the Prefab Asset also turns the original GameObject into a Prefab instance. Every change happening in the prefab, will happen in the instance as well.

• We can now delete **the AR Default Plane Object in the Hierarchy,** since we will only use the** Prefab.**

  

• On the XR Origin GameObject, go to the AR Plane Manager Script we added in Step 1. There is an empty Plane Prefab. Drag and Drop there the Prefab we created in the Assets.

  

You should now see the AR Default Plane Prefab placed as the Plane Prefab.

We can now visualize the AR detected planes in our device.

Let’s build the app and see if it works!

Build the AR App to test the AR Plane detection

Configure Build and Project Settings in Unity

1. Select Android in build settings

   • Go to File > Build Settings

   • Switch to the Android Platform - Go to player settings

   

   • If Unity requires you to download Android support in order to switch platforms, follow the link and install the required dependency.

   • Go to Player > Other Settings > Rendering

   • Make sure “Auto Graphics API” is unchecked. Change Color Space to “Linear”

   • Remove Vulkan from under Graphics APIs

   
   • Go to Player > Other Settings > Package Name. Create a unique app ID using a Java package name format. For example, use com.Princeton.AR
   • Go to Player > Other Settings > Minimum API Level. Select Android 7.0 ‘Nougat’ (API Level 24) or higher (For AR Optional apps, the Minimum API level is 14.)
   • Go to Player > Other Settings > Scripting Backend. Select IL2CPP instead of Mono.

   • Go to Player > Other Settings > Target Architectures. To meet the Google Play 64-bit requirement, enable ARM64 (64-bit ARM). Leave ARMv7 (32-bit ARM) enabled to support 32-bit devices

     

2. Configure Project Settings

   • Open Edit > Project Settings… and click on the XR Plug-in Management section. In the Android tab, enable ARCore.

     

3. Configure your phone to run the app

   • Plug your Android AR Phone into your computer, using the USB cable.

   • Make sure that your computer recognizes the device. You can check this in This PC>Devices and drives

     

   • If you don’t see it, try having the device unlocked when you plug it, and if USB options popup, select Use USB to: Transfer files on your phone screen.

     

   • Go to File>Build Settings

   • Respectively, you should be able to see your device detected in the Run Device drop-down Menu. You can click Refresh if you don’t see it right away.

     

4. Build and run the app

   • Click Build and Run. Select a folder and name your application as desired. This might take a bit of time. Keep the device unlocked, so that the application runs right away.

     

   • Move the phone slowly up and down, left and right, to start the detection.

   • After some time, you should be able to see this semi-transparent yellow material with an outline, indicating the scanned AR Planes.

Import a file into Unity from Rhino

Export file from Rhino

In Rhino, open your desired file. In our case it is a house form.

Note: Make sure you file is set to meters in Rhino.

• Move the x,y center of your geometry to 0,0,0. You can use the Move command, and and align it to the 0,0,0 plane. Have the x and y center at the origin, but the whole house should be above the ground plane.

  

Note: This step will help us have our model correctly oriented in Unity.

• Select the part(s) that you want to export

• Go to File>Export Selected

  

• Save as type .obj, in a folder that you can easily find e.g. another folder on your Desktop called “Seminar_Week_Assets”, and click “OK” in the OBJ Export Options dialog.

  

Note: Check that your file is created correctly in the folder.

Import file in Unity

First, we will learn how to import our model correctly into Unity, adjust its scale and position it correctly. In this example, Rhinoceros is a common CAD software, but you can use any preferred modeling software. Be sure to structure your model according to the materials you want to apply on it later. Unity understands a variety of file types, you can use OBJ and export each layer as a separate OBJ file.

Back in the existing Unity App, go to the Prefab folder we created before in the Assets (Project Window).

• Go to the Assets folder of the Unity Project on your computer (in the file explorer) , and copy paste both the .obj and the .mtl. You can make a new folder called ImportedObjects.

Note: Unity Projects are actually folders that contain other folders. So you can have full access on Prefabs, scripts, objects. When you put something in this folder, it updates in Unity as well.

• Inside the ImportedObjects folder, copy and paste both your .obj and the .mtl file that you exported from Rhino.

Note: Alternatively, you can make the new folder inside Unity and drag and drop both of the files.

• Back to Unity, you will see that the folder is also updated in the Project Window > Assets.

  

• Inside the folder, you should be able to see an icon of your house in the Prefabs. Click on it once and go to the Inspector Window. Enable Read/Write. Click “Apply”.

  

Create Prefab with imported Model

• Drag and drop the house Prefab into the Hierarchy. You will now see the object imported in your Scene.

• In the Inspector, make sure that the Transform, Position and Rotation are set to 0,0,0.

  

• To check that the Scale of the model we imported is correct. This is 1:1 in physical space. We can compare it with a 1,1,1 cube (1 x 1 x 1 meters) (Create>3D Object>Cube). If we see that the Cube is too big or too small, we have to scale our model. It is better to have an object that is small enough to fit in the phone camera.

• To scale our model, we can click on the house prefab once, go to the Inspector, and type in the amount of desired scale (e.g. if your model was in milimeters, use 0.001). Click Apply on the bottom right.

  

Note: Don’t forget to Save your Project from time to time! File > Save

Now that the scale of the object is correct, we can apply a Material of our choice. Let’s go back to Assets and make another empty Folder called Materials. Here we will be organizing our materials throughout the development.

Tip: You can rename an existing folder by clicking on it once, and pressing F2 on your keyboard.

• Inside the Materials folder, create a new Material (Right Click > Create > Material) Like the steps before, you can assign a color, or create multiple materials and assign them on your object.

• When you are finished, Drag and Drop the object GameObject back to the Assets>Prefabs, and name it as Building_Prefab_01. Today, we will use this Prefab to instantiate it in the AR Detected Planes.

  

Instantiate object

Now we can start the scripting base! The goal for today is to “Tap and Instantiate” the 3D object in our App, on the detected AR Planes.

Import the Instantiator Script

• In the Assets, Create a new empty Folder and name it “Scripts”. Here, we will be collecting all of the C# scripts we will be using.

• Go to the folder for this tutorial which you downloaded. Select the Instantiator.cs file, drag and drop the script into the Script folder we created in Unity. This is the base script, that we will use to build our application upon.

  

• Make an Empty Game Object and name it_ “Instantiator”. _Here we will assign our Instantiator script, where we manage the instances of the objects we create in the App.

  

• On the Instantiator GameObject, go to the Inspector and click on Add Component. Find and add the Instantiator script we imported in the previous steps.

  

Note: Always make sure that the Transform Values are set to Zero when creating a new GameObject.

• We see some defined variables that don’t have any object assigned to them. But first, let’s take a look at the code.

  

Overview of the code

• To open the script, you can either go to Assets>Scripts and double click, or on the component itself, and double click on the name.

  

  

Do these variables seem familiar? They are the ones that appear on the Inspector!

We can choose what kind of GameObject we need.

E.g. For the Camera, we only need the “Transform” component, to get information about its position and rotation in space.

Let’s drag and drop the objects we need.

• For the Selected Prefab, we can put the House Prefab we created, by dragging it from the Assets>Prefabs folder we created.

• For the AR Camera Transform, we will drag and drop the Camera Offset, which we can find under XR Origin in the Hierarchy.

  

Note: Make sure the Transform is set to 0,0,0 in the Inspector for the new object. Likewise, make sure it is set to zero for the AR Session and XR Origin GameObjects.

• The Inspector should look like this

  

Build App

• We are ready to build the App and test it!

  

Misc Information

1. MonoBehaviour is the base class from which every Unity script object derives its properties and methods (functions).
2. A class enables you to create your custom types by grouping variables of other types, methods, and events.
3. The API acronym stands for Application Programming Interface, which is a software intermediary that allows two applications to talk to each other.
4. Software Development Kits (SDKs) allow development for a variety of platforms. They contain the compiler and debugger and usually software frameworks. For example, Unity has the Android SDK, which allows you to build for the Android operating system.
5. Tags help you link GameObjects to variables in your scripts.